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Abstract 



Referencing a conceptual framework and the results of two previous investigations (that used 
concept mapping/pattern matching and qualitative interviewing methods), the present paper reports 
quantitative (survey) results to complement and generalize prior findings within the case organization 
studied. A scannable questionnaire survey booklet was developed using Teleform software. Scanned 
survey data were exported and analyzed. Based on an achieved sample of 280 (70% response rate) these 
analyses involved the computation of descriptive statistics as well as the construction of several scale 
variables corresponding to perceived training program outcomes, evaluation processes and uses. 
Reliability was examined using Cronbach's Alpha as a coefficient of internal consistency. Program 
participant and line management stakeholder groups were compared using independent t-tests and 
multivariate analysis of variance (MANOVA) for comparisons of stakeholder group scores on multiple 
item sets (i.e., dependent variables). Further analyses also examined relationships between perceived 
training outcomes and preferences for evaluation processes and uses by group. Three predictor and five 
criterion (scale) variables were constructed using the conceptual framework. The predictor variables were 
labeled customer satisfaction, product development, employee satisfaction. The criterion variables were 
labeled as evaluation purposes, process (data collection and participation/reporting), consequences 
(instrumental/conceptual and symbolic use). Stepwise multiple regression revealed several highly 
significant relationships (p < .001) between predictors (training results) involving employee and customer 
satisfaction and the criterion variables (training evaluation). The overall results suggest that — ^with a few 
notable exceptions — both managers and non-managers tended to view both the results and evaluation of 
training as being closely connected to employee satisfaction. These results are further discussed with 
written comments also collected from the survey. An expanded deployment of the survey is 
recommended beyond the case organization for future research. 
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A Quantitative Investigation of Stakeholder 
Variation in Training Program Evaluation 



As mainstream training evaluation practice continues to focus predominantly on demonstrating 
positive return on investment (ROI) at the course level from the singular stakeholder perspective of 
corporate management (who pay for courses), a growing body of literature suggests that, particularly 
within knowledge-intensive organizational settings, training evaluation would be better performed at the 
program level from multiple stakeholder perspectives (Altschuld & Zheng, 1995; Hackett, 1997; 
McLinden & Trochim, 1998; Stewart, 1999). A clearer understanding of stakeholder variation has 
several important implications for the theory, practice of training program evaluation. For example, (1) 
Can any single training evaluation model equally serve all organizational constituencies? (2) Which 
stakeholder group(s) or individual(s) is/are best positioned to plan and execute the evaluation, and further 
to interpret, and act, on its findings? (3) Whose ends are served by large-scale corporate training and the 
results of such typically highlighted in formalized training evaluations? 

This paper reports the results of a survey designed to investigate variation in stakeholder 
perceptions of training results and evaluation within the context of a high-technology product 
development firm (case organization). The results reported here are derived directly from the author's 
dissertation research as a multiple-method study organized into three phases. Details of the conceptual 
framework developed and the results of from the prior two research phases have been reported and are not 
re-stated here (see Michalski, 1997; Michalski, 1998; Michalski & Cousins, in press). The objectives of 
the overall dissertation study were to answer the following two research questions: (1) To what extent do 
training stakeholder groups differ in their perceptions about the importance and evaluation of training 
program results? What are the key dimensions of divergence? (2) Do stakeholder views about training 
program evaluation depend on the organizational results they perceive for the program? If so, in what 
ways? 



Purpose 

The purpose of the survey was to further address research question two (above) regarding 
stakeholder views of training evaluation in relation to their perceptions of training results. The survey 
involved managers (sponsors) and non-managers (trainees) in the case organization, a division of a large 
network and telecommunications development firm headquartered in Ontario, Canada. The central 
analyses performed here were concerned with the relationship between three predictor variables 
developed to describe stakeholder perceptions of training results and five criterion variables developed to 
describe stakeholder perceptions of training evaluation. The three predictor variables developed 
described training results in terms of customer satisfaction, product development, and employee 
satisfaction. The five criterion variables derived described training evaluation in terms of evaluation 
purposes, data collection, participation and involvement, instrumental-conceptual use, and symbolic 
evaluation use. 

Beyond several marginal differences revealed between the groups regarding their respective 
perceptions of training results and training evaluation, the key concern relative to research question two is 
which (if any) of the predictor variables (training results) are related to the criterion variables describing 
stakeholder perceptions of training evaluation. The following sections provide a progressive description 
of the survey development and analyses performed. 

Sample 

The training participant (non-manager) population consisted of approximately 700 engineers, 
designers, and technicians, while the manager population consisted of approximately 100 managers in the 
case organization. Based on researcher experience with the organization it was known that managers 
generally respond to surveys at a proportional rate of three to one relative to engineering staff, an 
appropriate proportional sample was prepared using simple random sampling principles (see, e.g.. Miller, 
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1994). Also, because training is equally available to all members of all organizational subdivisions, 
sampling stratification was considered but deemed unnecessary. From the organizational population a list 
of names (each accompanied by departmental and job classification information to distinguish managers 
from non-managers) was randomly selected using the most current internal employment record 
information available from the company’s human resources group. 

Imnlementation and Response 

Several authors have emphasized the importance of achieving sufficient response rates in surveys. 
For example, Fink (1995a) discussed the problems associated with non-response at both the survey and 
item level. Furthermore, Bourque and Fielder (1995) emphasized the importance of the many details of 
survey preparation and administration. They discussed many of these in terms of obtaining a valid and 
reliable sample in connection with achieving a sufficient response rate. This was a particular concern for 
the present research because the population sampled is very routinely polled and surveyed using all 
varieties of techniques (mail, phone. E-mail, web, even live solicitation). Hence, extreme care and effort 
was taken to maximize the response rate of this survey. This included careful considerations of details 
such as questionnaire objectives; general length and format; item wording, length, and format; use of 
clear and sufficient instructions and contact information; pilot testing; cover letter; and follow-up 
reminders. 

A total of 415 surveys were mailed via interoffice mail to randomly selected individuals from the 
case organization population sampling frame consisting of 610 names. The list of individuals for the 
sample was obtained from the host organizations human resources database and was sorted to show the 
names of regular, full-time employees with job band classifications five through ten (inclusive). It 
included a total of 458 non-managers (bands 5 and 6) and 152 managers. This population of knowledge 
workers consists mostly of telecommunications engineers, programmers, designers, as well as project and 
line managers. Because 15 individuals from the original list were later determined to be invalid because 
they had left the division (or company), or were otherwise unavailable to participate (for example due to 
vacation or leave of absence) a final total of 400 surveys were sent to individuals including 100 managers 
and 300 non-managers. 

An achieved sample of 280 (70% response rate) was obtained consisting of 60 managers and 220 
non-managers. This represents a proportion of 21% managers and 79% non-managers who responded to 
the survey. Careful random sampling is credited for attaining an achieved sample with demographic 
proportions closely comparable to their respective (total) population proportions. For example, the 
sampling frame included 25% managers and 75% non-managers as determined by job band. 

Instrument 

A four page questionnaire survey booklet was developed in view of research question two and the 
results obtained in the prior two phases of research (Michalski & Cousins, in press; Michalski, 1998). 

The final conceptual cluster titles from the phase 1 investigation were used as a starting point to identify 
six initial question categories for part 1 of the survey regarding perceptions of training results beneficial 
to the organization. These cluster titles reflected several common themes, such as customer and 
employee satisfaction, perceived by both line managers (training sponsors) and non-managers (trainees). 
They were used to derive six distinctive categories for survey part 1 scale variable construction as 
discussed later in this paper. 

The average importance ratings assigned by each of the two groups to the phase 1 statements was 
further used to identify training results within each of the six areas included in survey part 1. To do this a 
simple differential was calculated by arithmetically subtracting the average trainee ratings from the 
average sponsor ratings for each of the 100 statements. These calculations (performed using a common 
spreadsheet software application) resulted in a column of 100 difference scores (deltas) each 
corresponding to a phase 1 statement. A descending bubble sort was performed on this column of 
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differences to identify rating dissimilarities between the groups^ As simple differences, these delta 
values were both positive and negative. A large positive value indicated a statement that was rated (on 
average) higher by the sponsor group relative to the trainees. A large negative value indicated the 
opposite relationship (a difference at or close to zero indicated minimal difference). For example, for 
statement #23 (support key performance plan of organization) the sponsor average importance rating (as a 
training result) was 4.08 (very important). The average importance for this same statement as assigned by 
trainees was 2.77 (somewhat to moderately important). The difference between these two averages was 
1.31 indicating that sponsors generally rated the statement higher in average importance relative to 
trainees. A similar calculation for statement #8 (training is integrated with university- industry interaction 
programs) yields a difference of -0.93 indicating that trainees generally considered the statement (training 
result) more important relative to sponsors. In providing an indication of statements rated differently by 
managers and non-managers, this process provided some objective guidance to formulate and balance the 
representation of the 20 items finally developed for part 1 of the survey. A minimum of three items per 
scale variable were planned. 

Results from a prior qualitative investigation (Michalski, 1998) were also employed to formulate 
the items contained in part 2 of the survey. This survey section focused on respondent perceptions related 
specifically to training evaluation. The first three items were concerned with respondent perceptions 
related to the purpose of training evaluation. Items 4 through 8 focused on stakeholder involvement. 

Items 9 through 13 sought to capture respondent views about the type of data required for training 
evaluation. Items 14 through 17 were concerned with views related to evaluation reporting audiences. 

The last nine items (18 through 26) were focused on the consequences of training program evaluation 
especially in terms of conceptual, instrumental, and symbolic uses. Similar to the three sections of the 
semistructured interview guide developed for phase 2 (Michalski, 1998), the 26 items comprising survey 
part 2 sought to capture data related to stakeholder view of the purposes, processes, and consequences of 
training evaluation. 

At the request of the management group that approved deployment of the survey ten items (part 
3) were also included to measure respondent general satisfaction with the existing training program. This 
purpose was mentioned in the cover letter that accompanied the survey. While the part 3 data was not 
used for the analyses reported here the pooled analysis of these ten items was made available to 
respondents as promised in the cover letter. Because by far most respondents who completed parts 1 and 
2 of the survey also completed part 3, and because part 3 was physically located on the last page of the 
survey form (just before the demographic section at the end) there was little concern (and no evidence) 
that the ten part 3 questions affected responses of the first two sections. 

Technically and from an analytical perspective, the development and analysis plan of this 
instrument was also informed by prior survey research related to evaluation practice. Referencing 
Cousins, Donohue, and Bloom, (1996) the scale variables were constructed as linear combinations of 
Likert type item sets using the research questions and conceptual framework. As described below in the 
analysis section, relationships among variables were examined using bivariate and multivariate 
procedures. Intercorrelation matrices using Pearson correlation were examined for variable set patterns. 
Stepwise multiple regression was used to account for variation in criterion variables and to assess the 
relative uniqueness and magnitude of contributions by predictors. Repeated measures multivariate 
analysis of variance (MANOVA) was used to test for differences between groups. 

The questionnaire was developed following general techniques of good survey design (see, e.g., 
Fink, 1995 b,c,d; Mangione, 1995; Rodeghier, 1996; Rosenberg, 1968). Realizing that the survey 
instrument was being developed primarily in relation to the research project as a new instrument and, 
therefore, does not enjoy long history to compare certain aspects of reliability and validity, these areas 
were, nevertheless, considered. For example, the instrument’s stability (test-retest reliability) and 



* Although the spreadsheet used to produce the calculations is not provided in the thesis, the data in Appendix H can 
be used to calculate these delta values. 
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alternate-form reliability are obviously not known due to its single form and instance of administration. 
The homogeneity of items and scale variables was possible to assess using Cronbach’s alpha. This is 
discussed further shortly. 

The content, face, criterion, and construct validity of the survey were also considered, but again 
the initial administration of the instrument limits extensive conclusiveness here. As noted the content for 
the survey was derived from the conceptual framework with reference to the literature review performed. 
Care was taken to ensure a high quality and highly readable form to enhance face validity. Due to the 
research nature of the instrument, criterion validity (predictive and concurrent) could not be conclusively 
established. According to Fink (1995a) construct validity (convergent and discriminant) is established 
experimentally to demonstrate that a survey distinguishes between people who do and do not have certain 
characteristics. This can be accomplished in at least two different ways both of which rely upon existing 
(valid) instruments for comparison or well-developed theory. Again, due to the dearth of material 
available in either of these categories specifically related to the research undertaken here, conclusive 
claims about the construct validity of the instrument cannot be made. Rather, the research project as a 
whole might be viewed as a precursory contribution to efforts aimed at developing future versions of the 
instrument. 

Pilot testing of the instrument was performed by mailing the form to a total of six members of the 
target population who agreed to complete it and provide feedback for improvement. These individuals 
did not participate in the survey proper. Comments from these individuals were used to ensure the 
appropriate level of language and usage of terms understandable to the target population. Face validity 
was also examined by discussing the overall look, readability, time and ease of completion. 

The survey form included two primary data collection sections with a total of 46 items related to 
respondent perceptions of general training results (part 1 containing 20 items) and training program 
evaluation (part 2 containing 26 items). A five-point, balanced, attitudinal scale was used for all items 
(strongly disagree to strongly agree). As mentioned the third section was included only to collect 
participant satisfaction data relative to the current training program. Because they are outside the scope 
of the present study, the results of these ten survey questions are not presented or discussed in connection 
with the present study. Similarly, as taken from the fourth and final survey section, only the demographic 
data about respondent job classification (i.e., manager or non-manager) were used in the current study. 

To maximize the accuracy of data entry, the final instrument was produced as a scanable form 
using the software application called Teleform (version 5.4). This software produces survey forms from 
which data can be scanned directly into a number of database formats (e.g., SPSS) using a fax machine or 
flatbed scanner. 

Implementation and Deployment 

The survey deployed was both random and anonymous. To encourage participation and 
ameliorate the overall response rate, prenotification of all potential respondents was accomplished using 
standard postcards distributed through interoffice mail 1 week before the survey was mailed. Similar 
postcards were also sent as reminders 2 weeks after the survey package was mailed. 

The survey package contained three items including (1) a cover letter, (2) the questionnaire form, 
and (3) a self-addressed interoffice mail envelope. A recommended return period of 1 week was 
suggested in the cover letter. To accommodate travel and vacations, a total period of 4 weeks was 
allowed for responses. The cover letter accompanying the survey was drafted by the principle researcher 
and co-signed on his behalf by the vice president responsible for training in the organization. The vice 
president’s signature was used because (unlike that of the researcher) his name was well known among 
the potential respondents within the case organization. Such cover letter name familiarity has been 
shown to be beneficial in boosting survey response rates (Bourque & Fielder, 1995). The cover letter and 
complete survey form are shown in Appendix A. 
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Results 

As shown in Appendix B survey response was also broken down demographically by the 
experience level (time in company), product development responsibility, and job category of the 
respondents. Nearly half (47%) of all respondents indicated 1 to 5 years of experience in the company 
followed by 20% who indicated 10 to 20 years experience. Proportionally smaller percentages of 
respondents indicated 5 to 10 years (14%), less than 1 year (14%), or over 20 years (5%) of experience 
respectively. Approximately 44% of all respondents indicated a primary affiliation with the optical 
carrier (OC) 192 development group. Another 18% identified an affiliation with the OC-48 group while 
OC-12 was identified by 14% and OC-3 by 13% of respondents. The category “other” product group was 
indicated by 1 1% of respondents who wrote in affiliation such as TNUI (transport node user interface), or 
various combinations of the OC development groups among others. Nearly 90% of respondents indicated 
Job functions of either software (60%) or hardware (29%) as their primary job function. Verification was 
indicated by 2% and captive office by 1% of respondents. The “other” job category was selected by 8% 
of respondents and included write-in functions such as “project management”, “design field support”, 
“customer support”, “characterization”, “program office”, and “design advisor”. All product groups 
were represented well by both managers and non-managers. For example, manager response by product 
group ranged from 15% (OC-12) to 40% (other). The “other” category included write-in job-roles such 
as “project manager” that correspond to specialized managerial roles. 

Part 1: Training results 

Part 1 of the included 20 items pertaining to stakeholder perceptions of training results. As 
mentioned, these items were developed based on findings in the previous two studies with reference to the 
conceptual framework developed. The item-level results by stakeholder group for each of the 20 part 1 
items are first examined followed by a similar examination of the items in survey part 2. 

As mentioned above, all items were scored using a five-point scale: l=strongly disagree, 
2=disagree, 3=neither agree or disagree, 4=agree, 5=strongly agree. Table 1 displays each of the items 
from survey part 1 along with group comparisons of item means, standard deviations, and numbers of 
respondents in both stakeholder groups. As shown, missing data problems were minimal with most items 
answered by all of the managers (n=60) and non-managers (n=220) who returned surveys. 

Item means (M) ranged from 2.03 (managers Only job-specific training should be provided to 
employees) to 4.40 (managers The availability of high-quality training leads to increased employee 
satisfaction). Item standard deviations ranged from .61 (non-managers The availability of high-quality 
training leads to increased employee satisfaction) to 1.04 (managers The main purpose of training should 
be to improve customer satisfaction). 

Item-level group means were also compared using independent samples t-tests. The null 
hypothesis is that there is no difference between average ratings by managers and non-managers. 

The alternative is that there is a difference. By selecting an alpha level of significance of .05 as the 
probability of committing a Type I error (rejecting the null hypothesis when it is true) significant 
differences (p< .05) between managers and non-managers were found in only three of the 20 items. In 
all three cases non-managers agreed more strongly than managers with the statements which had to do 
with training results respectively in terms of anticipating customer requirements, product innovation, and 
employee professional development. For the item “Effective training should help designers to anticipate 
customer requirements” the average ratings by non-managers (M=3.65) was significantly greater (p = 

.017) than those of managers (M=3.33). Non-managers also rated the item “Effective training ultimately 
leads to product innovation” significantly higher (M=3.67; p = .001) than managers (M=3.22). Similarly, 
for the item “The main purpose of training should be to develop employees professionally” the average 
rating by non-managers (M=4.16) was significantly greater (p < .001) than that of managers (M=3.77). 
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Part 2; Training Program Evaluation 

The same five-point scale used for part 1 was also used to score items in part 2. Table 2 displays 
each of the items from survey part 2 along with the item mean, standard deviation, and number of 
respondents by stakeholder group. While missing data was similarly not excessive, the number of 
individuals who rated all items in part 2 was obviously less than that of the previous section on training 
results. Generalizations can be made as to the possible reasons for this (e.g., respondent fatigue), 
however, the written comments (discussed later) indicate that a substantial number of respondents 
considered the items in this section as somewhat technical and beyond their specialty area or realm of 
expertise. Nevertheless, each item in part 2 was rated by a minimum of 55 managers and 213 non- 
managers, with most items being rated respectively by at least 57 and 215 individuals. Item means 
ranged from 2.14 (managers — Training program evaluation results should be used mainly to comply with 
quality standards such as ISO audits) to 4.14 (managers — Trainees should be actively involved in 
performing the training program evaluation). Item standard deviations ranged from .47 (managers 
Training program evaluation results should be used mainly to develop new knowledge about program 
effects) to .99 (managers — External evaluation experts should be actively involved in performing the 
training program evaluation). 

Item-level group means were also compared using independent samples t-tests. Significant 
differences (p < .05) between managers and non-managers were found in only two items. Managers 
agreed (M=4.10) more strongly than non-managers (M=3.90) that “Training program evaluation results 
should be used mainly to determine if the program is meeting its goals” (p = .047). However, non- 
managers provided a higher rating (M=2.84; p < .001) to the statement “Training program evaluation 
results should be used mainly to comply with quality standards such as ISO audits” than managers 
(M=2.14) who more clearly tended to disagree with the statement. 

Several multiple-item scale variables were also constructed as linear combinations of item 
average scores. In addressing the second research question about whether stakeholder views of training 
program evaluation depend on the results they perceive for the program, all items from both part 1 and 
part 2 were initially grouped respectively in terms of specific subcategories of training results and 
evaluation purposes, processes, and consequences. This process ultimately resulted in the definition of 
three predictor variables for training results and five criterion variables for training evaluation. The 
details of predictor and criterion variable construction are described next. 

Training Program Result Predictor Variables 

Referencing previous results (Michalski & Cousins, in press), all items in part 1 of the survey 
were organized conceptually into subcategories of training results. As shown in Table 3 the items were 
grouped initially according to the following six conceptual subcategories of training results: (1) business 
and market, (2) customer satisfaction, (3) product development, (4) employee productivity, (5) training 
efficiency, and (6) employee satisfaction. While each of these initially contained a minimum of three 

items to allow for the calculation of reliability coefficients (Cronbach’s alpha^) unacceptably low 
reliability coefficients were revealed using these sub-categories. The coefficients for each of the 
categories were as follows: business and market (a =.53), customer satisfaction (a =.59), product 
development (a =.46), employee productivity (a =.40), training efficiency (a = 48), and employee 
satisfaction (a =.61). Although Hinkin (1998) stated that a coefficient alpha value of .70 is considered 
large for exploratory measures, the original reliability coefficients were considered too low. To improve 
this situation items from part 1 were regrouped into three conceptually broader subcategories describing 
training results in terms of (1) customer satisfaction, (2) product development, and (3) employee 
satisfaction. As shown in Table 4 this regrouping did substantially improve reliability among the 
variables and so these were developed and used for subsequent analysis. 



2 Cronbach's Coefficient Alpha (a) is a general form of the Kuder-Richardson (K-R) 20 formula used to estimate the 
internal consistency of items when they are not scored dichotomously, see Borg and Gall (1989). 
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Table 1 . Item-Level Results by Stakeholder Group for Survey Part I 





Survey Items: Training Program Results 


Stakeholder 

Group 


n 


M 


SD 


• 


An effective training program attracts or retains 


manager 


59 


3.34 


.94 




the most talented employees. 


non-manager 


220 


3.53 


.98 


• 


Effective training ultimately improves business 


manager 


60 


4.35 


.66 




profitability. 


non-manager 


219 


4.37 


.66 


• 


The main purpose of training should be to 


manager 


60 


2.90 


1.04 




improve customer satisfaction 


non-manager 


220 


2.95 


1.00 


• 


Effective training should directly support our 


manager 


60 


3.43 


.85 




customer’s business objectives 


non-manager 


220 


3.41 


.86 


• 


Effective training should help designers to 


manager 


60 


3.33* 


.97 




anticipate customer requirements 


non-manager 


220 


3.65* 


.89 


• 


Effective training should improve employee 


manager 


60 


4.03 


.64 




understanding of product development processes 


non-manager 


219 


3.90 


.67 


• 


Effective training ultimately improves employee- 


manager 


60 


3.62 


.87 




customer relations 


non-manager 


220 


3.43 


.81 


• 


The main purpose of training should be to 


manager 


60 


3.83 


1.01 




improve employee productivity 


non-manager 


219 


3.98 


.94 


• 


The availability of high-quality training leads to 


manager 


60 


4.40 


.74 




increased employee satisfaction. 


non-manager 


220 


4.39 


.61 


• 


Effective training ultimately leads to product 


manager 


60 


3.22** 


1.01 




innovation 


non-manager 


219 


3.67** 


.90 


• 


Only Job-specific training should be provided to 


manager 


60 


2.03 


.92 




employees 


non-manager 


220 


2.08 


.95 


• 


Effective training should result in the immediate 


manager 


59 


3.36 


.94 




use of knowledge on the Job 


non-manager 


220 


3.31 


1.02 


• 


Training is the best way to make new employees 


manager 


60 


3.85 


.88 




productive as quickly as possible 


non-manager 


220 


3.90 


.94 


• 


Employee pay and employee training are directly 


manager 


60 


2.22 


.92 




related 


non-manager 


220 


2.43 


.83 


• 


The main purpose of training should be to 


manager 


60 


3.77** 


.91 




develop employees professionally 


non-manager 


219 


4.16** 


.71 


• 


Employee promotion and employee training are 


manager 


59 


2.54 


.95 




directly related 


non-manager 


220 


2.70 


.98 


• 


Product quality can be improved substantially 


manager 


60 


4.32 


.68 




through employee training 


non-manager 


218 


4.19 


.86 


• 


All training provided to employees should have a 


manager 


60 


3.45 


1.00 




positive “Return on Investment” (ROI) 


non-manager 


219 


3.50 


.95 


• 


Training program objectives should be derived 


manager 


59 


3.36 


.98 




directly from organizational business objectives 


non-manager 


219 


3.14 


.95 


• 


Product Time to Market (TTM) can be reduced 


manager 


60 


3.48 


.98 




substantially through employee training 


non-manager 


219 


3.58 


.84 



Note; group mean eomparison using independent samples t-test (alpha = .05) 
♦ p<.05 ♦♦ p<.001 
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Table 2. Item-Level Results by Stakeholder Group for Survey Part II 





Survey Items: Training Program Evaluation 


Stakeholder 

Group 


n 


M 


SD 


• 


The main purpose of training program evaluation is to 


manager 


57 


4.07 


.62 




improve the program 


non-manager 


216 


4.08 


.62 


• 


The main purpose of training program evaluation is to judge 


manager 


56 


3.80 


.67 




whether the program should be continued 


non-manager 


216 


3.94 


.69 


• 


Training specialists should be actively involved in 


manager 


58 


3.72 


.79 




performing the training program evaluation 


non-manager 


217 


3.88 


.69 


• 


Trainees (training recipients) should be actively involved in 


manager 


58 


4.14 


.54 




performing the training program evaluation 


non-manager 


217 


4.04 


.69 


• 


Training sponsors (line management) should be actively 


manager 


58 


3.74 


.81 




involved in performing the training program evaluation 


non-manager 


217 


3.69 


.75 


• 


External evaluation experts should be actively involved in 


manager 


58 


3.28 


.99 




performing the training program evaluation 


non-manager 


216 


3.40 


.89 


• 


Questionnaire data from training participants is essential in 


manager 


59 


3.63 


.79 




training program evaluation 


non-manager 


216 


3.74 


.76 


• 


Interview data from training participants is essential in 


manager 


59 


3.61 


.70 




training program evaluation 


lion-manager 


216 


3.40 


.82 


• 


Focus group data from training participants is essential in 


manager 


59 


3.44 


.73 




training program evaluation 


non-manager 


216 


3.40 


.78 


• 


Knowledge test data from training participants is essential 


manager 


59 


3.47 


.97 




in training program evaluation 


non -manager 


216 


3.33 


.88 


• 


All training program evaluation results should be reported 


manager 


59 


4.05 


.63 




directly to training specialists 


non-manager 


217 


3.93 


.65 


• 


All training program evaluation results should be reported 


manager 


59 


3.61 


.89 




directly to trainees (training recipients) 


non-manager 


217 


3.63 


.77 


• 


All training program evaluation results should be reported 


manager 


59 


4.03 


.76 




directly to training sponsors (line management) 


non-manager 


217 


3.83 


.76 


• 


Training program evaluation results should be used mainly 


manager 


56 


3.89 


.73 




as a basis for future decisions about the program 


non-manager 


215 


3.98 


.67 


• 


Training program evaluation results should be used mainly 


manager 


55 


3.82. 


.47 




to develop new knowledge about program effects 


non-manager 


213 


3.77 


.71 


• 


Training program evaluation results should be used mainly 


manager 


57 


2.14** 


.93 




to comply with quality standards such as ISO audits 


non-manager 


215 


2.84** 


.86 


• 


Training program evaluation results should be used mainly 


manager 


57 


3.25 


.83 




to routinely monitor program activities 


non-manager 


215 


3.36 


.85 


• 


Training program evaluation results should be used mainly 


manager 


57 


3.82 


.85 




to determine trainee satisfaction with the program 


non-manager 


215 


3.87 


.74 


• 


Training program evaluation results should be used mainly 


manager 


57 


3.25 


.97 




to determine manager satisfaction with the program 


non-manager 


214 


3.24 


.81 


• 


Training program evaluation results should be used mainly 


manager 


58 


4.10* 


.58 




to determine if the program is meeting its goals 


non-manager 


215 


3.90* 


.72 


• 


Training program evaluation results should be used mainly 


manager 


58 


3.09 


.88 


Note: 


to assist trainees in selecting courses 

erouD mean comnarison using indenendent t-tpct /'alnha = r 


non-manager 


213 


3.34 


.92 



'P<.05 **p<.001 
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Table 4. New Predictors— Training Program Results (15 out of 20 original items used) 
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It should be noted that, in an effort to obtain better scale variable reliability, a departure was taken 
from the conceptual framework and an exploratory factor analysis (principle components with 
varimax rotation; factor extraction using Eigenvalues > 1; factor loadings > .40) was also 
performed on all part 1 items. This yielded five factors which accounted for 52% of the variance. 
However, because only one factor (factor 1) yielded a potential scale variable with a reliability in 
excess of .7, and because the remaining potential scale variables (defined by the remaining 
factors) were conceptually unrelated to each other, the use of factor analysis to construct scale 
variables was abandoned in favor of the original conceptual framework described. Appendix C 
contains the detailed results of this factor analysis. 

Training Program Evaluation Criterion Variables 

Referencing the conceptual framework and qualitative results (Michalski, 1998), a set of 
predictor variables related to training evaluation was also developed from the part 2 survey items. 
Similar to the process used to derive the part 1 scale variables, the criterion variables were also 
organized into conceptually sound subcategories describing stakeholder perceptions of the 
purposes, processes (data collection, participation and involvement) and consequences of training 
evaluation. Table 5 shows the initial item groupings by subcategoiy. As shown training 
evaluation purposes are described in terms of formative and summative categorizations. 

Evaluation processes are divided into three subcategories related to (1) stakeholder involvement 
in the performance of the evaluation, (2) data collection, and (3) reporting audience. 

Additionally, eight items were included to describe various instrumental, conceptual, and 
symbolic consequences of evaluation. The reliability coefficients initially obtained for each item 
subcategory were as follows; purposes (a =.65), processes involvement (a =.59), data (a =.66), 
reporting (a =.56); consequences (all items combined, a =.75). Again, adhering to the original 
conceptual framework, these groupings were slightly re-organized to improve reliabilities and 
construct the criterion variables shown in Table 6. 

As shown, the criterion variables developed to describe training evaluation include (1) 
evaluation purposes, (2) data collection, (3) participation and involvement, (4) instrumental- 
conceptual use, (5) symbolic use. The reliabilities for these variables fall in the range of .65 to 
.69. Similar to the process used to analyze the part 1 items, a factor analysis (principle 
components with varimax rotation; factor extraction using Eigenvalues > 1; factor loadings > .50) 
was also performed for the part 2 survey items. Of the seven factors extracted reliability 
coefficients ranged between .61 and .42. Because of these low reliabilities and because the items 
composing the factors were conceptually unrelated, factor analysis was again abandoned (in favor 
of the conceptual groupings described). The factor analysis details for the part 2 items are also 
displayed in Appendix C. The next section describes the zero-order intercorrelation relationships 
revealed among the scale variables derived. 
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Relationsliins Among Scale Variables 

Table 7 shows the zero-order intercorrelations among the scale variables constructed to 
describe stakeholder views of training results and evaluation. This table shows the five criterion 
variables (variables one through five) presented first followed by the three predictors (variables 
six through eight). Significant (p < .05) to highly significant (p < .001) relationships were 
observed among all values. As shown by the cell values in the first five intersecting rows and 
columns, intercorrelations among the five criterion (dependent) variables are all positive ranging 
from .18 to .58. All positive intercorrelations, ranging from .44 to .57, were again observed 
among the predictor variables (rows and columns six through eight). The moderate size of most 
coefficients suggests that the scale variables are measuring different constructs. Also, because 
these values are moderate, assumptions regarding multicolinearity are not likely to have been 
violated. Additionally, as shown by the cell values in the intersections of columns one through 
five with rows six through eight, all intercorrelations between the predictor and criterion variables 
are positive (ranging from .15 to .29). These values suggest a positive relationship between group 
perceptions about training results (predictor) and those related to training evaluation (criterion 
variables). 

Table 7. Zero-order Intercorrelations among criterion and predictor (scale) variables 
(Pairwise deletion of missing data, N= 270 to 280) 



Variable^ 


1 


2 


, 3 


4 


5 


6 7 


8 


1. Evaluation 


— 














Purposes 
















2. Data Collection 


29*** 


- 












3. Participation and 


35*** 


.46*** 


„ 










Involvement 
















4. Instrumental- 


36*** 


33*** 


41*** 


__ 








Conceptual Use 
5. Symbolic Use 


.18** 


27*** 


37*** 


58*** 


- 






6. Customer 


j7** 


J7** 


26*** 


25*** 


24*** 


__ 




Satisfaction 
















7. Product 


.16** 


19** 


24*** 


17** 


.15* 


^7*** 




Development 
8. Employee 


.18** 


21 *** 


32 *** 


29*** 


23*** 


44*** .48*** 





Satisfaction 



“ Scale variables 1 through 5 are dependent (criterion) variables; 6 through 8 are independent (predictor) variables. 

*p<.05 **p<.01 ***p<.001 
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Analysis of Group Differences 

A MANOVA^ procedure was performed to test for differences between groups regarding 
views of training results using the three predictor variables constructed. A marginal difference 
between the groups was revealed (p = .055) by Hotelling’s multivariate F (2.563; ^=3; 276). 
While this multivariate result was marginal, as shown in Table 8 separate univariate tests 
revealed that the only significant effect was associated with the variable “employee satisfaction” 
(F = 7.05, E = . 008). 



Table 8. Predictor Variable Univariate Tests 



Variable 


ss 


df 


MS 


F 


P 


Customer Satisfaction 


1.31 


1 


1.31 


.20 


.655 




1816.63 


278 


6.54 






Employee Satisfaction 


49.46 


1 


49.46 


7.05 


.008 




1950.32 


278 


7.02 






Product Development 


6.66 


1 


6.66 


.67 


.415 




2781.12 


278 


10.00 







The F tests the effect of manager based on the linearly independent pairwise comparisons 
among the estimated marginal means. 



Item-level t-tests on the five statements composing this variable revealed a significant difference 
(e 5 .001) for only a single statement (1.8) on employee professional development. Based on the 
five-point scale used (1 = strongly disagree through 5 = strongly agree) non-managers agreed 
more strongly (M = 4.16) than managers (M = 3.77) with the statement: “The main purpose of 
training should be to develop employees professionally.” 

Of the remaining two predictor variables, “customer satisfaction” and “product 
development” each contained one statistically significant item based on the item-level t-tests 
discussed. Non-managers agreed more strongly (M = 3.65) than managers (M = 3.33) with the 
statement “Effective training should help designers to anticipate customer requirements” in the 
customer satisfaction variable. Non-managers also agreed more strongly (M =3.67) than 
managers (M =3.22) with the statement “Effective training ultimately leads to product 
innovation” in the product development variable. 

The MANOVA procedure was also used to examine differences between groups 
regarding views of training evaluation using the five criterion variables constructed. Somewhat 
greater difference between groups was revealed (p = .022) by Hotelling’s multivariate F (2.691; 
^= 5; 264). As shown in Table 9 separate univariate tests revealed only the variable describing 
“Instrumental-Conceptual Use “ to be significant (F = 5.83, p = .016). 



^ According to SPSS, Inc. (1997) both MANOVA and GLM provide generalized procedures for analysis of 
variance and covariance. The major distinction is that GLM uses a non-full-rank or overparameterized 
indicator variable approach in linear models instead of the full-rank reparameterization approach used in 
MANOVA. For the current analysis both procedures yielded identical results. 
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Table 9. Criterion Variable Univariate Tests 



Variable 


ss 


df 


MS 


F 


P 


Evaluation Purposes 


1.97 


1 


1.97 


1.51 


.221 




349.80 


268 


1.31 






Data Collection 


7.84 


1 


7.84 


1.55 


.215 




1359.92 


268 


5.07 






Participation and Involvement 


2.38 


1 


2.38 


.26 


.614 




2502.67 


268 


9.34 






Instrumental-Conceptual Use 


52.33 


1 


52.33 


5.83 


.016 




2406.77 


268 


8.980 






Symbolic Use 


5.77E-02 


1 


5.768E-02 


.029 


.865 




530.24 


268 


1.98 







The F tests the effect of manager based on the linearly independent pairwise comparisons among 
the estimated marginal means. 



Significant differences were found at the item-level in two of the six items composing this 
variable. As discussed in the item-level analysis, both statements that showed significant 
differences between managers and non-managers were found in this scale variable. In the first of 
these, managers agreed more strongly (M = 4.10) than non-managers (M = 3.90) with the 
statement “Training program evaluation results should be used mainly to determine if the 
program is meeting its goals.” However, non-managers agreed much more strongly (M = 2.84) 
than managers (M = 2.14) with the statement “ Training program evaluation results should be 
used mainly to comply with quality standards such as ISO audits.” As described later, several of 
the written comments also support these differences. For example, these comments suggest that 
managers tend to perceive training results more in terms of the extent to which they help to 
achieve certain management objectives, hence, they favor instrumental-conceptual modes of 
evaluation utilization that support this view. 

Perceptions of evaluation in terms of training results. 

Stepwise multiple regression was also used to further explore the relationship between 
stakeholder views of training results and evaluation. As shown in Table 10 the five criterion 
variables were regressed on the three predictors in separate models. As indicated by the values 
for R , a relatively low amount of variability was explained by each of the models. However, 
several significant relationships were revealed between stakeholder views of training results in 
terms of both employee and customer satisfaction. As shown, the predictor labeled “employee 
satisfaction” was retained and most significant in all models constructed. This outcome suggests 
that stakeholder group perceptions of evaluation are influenced by their perceptions of training 
results particularly in terms of employee satisfaction (see research question two). 
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Discussion and Limitations of Quantitative Results 

Perhaps not surprisingly in view of the results obtained in the previous two studies, only 
moderate variation was found among the training client group stakeholders. The variation that 
was found, however, supports the general predictions made regarding manager and non-manager 
emphasis in terms of both the results and evaluation of training. While both groups related 
training results closely with overall employee satisfaction, significant differences found in both 
the univariate and multivariate tests suggest that employees perceived training results more in 
terms of their own professional development than did their line manager counterparts. Both 
groups of training evaluation were shown to be related to their views of training results 
marginally in terms of employee and customer satisfaction. 

Before attempting to generalize these results further, however, several limitations should 
be recognized. First, the relatively low reliability exhibited among the scale variables is 
considered a notable limitation. In performing these analyses it was found that various other item 
combinations (e.g., using factor analysis) could indeed produce scale variables with higher 
reliability coefficients. However, as described such combinations tended to deviate profoundly 
from the conceptual framework and research questions thereby producing “variables” that largely 
defied interpretation. For example, grouping the part 1 items related to “pay” and “promotion” 
does indeed yield a high coefficient alpha (.78), however, this is the case simply because there is 
no significant difference between managers and rion-managers in their mild disagreement that 
training is directly related to these. Simply stated, neither group indicated a belief that there is 
some direct proportionality between training and one’s pay or promotion status in the 
organization. 

The second limitation to note is the relatively low coefficients obtained in the 
regression analysis. The ability for the models defined to describe more variation could well be 
related to the relatively low reliabilities for the scale variables. Hence, while the results obtained 
suggest that stakeholder group perceptions of training program evaluation may be moderately 
dependent on the training results they perceive in terms of employee satisfaction, further 
investigation could only improve our understanding of this apparent relationship. 

A third limitation is that these results were obtained within a single organization. Any 
generalizations to be made from the results would have to be based on a broader deployment of 
the survey instrument in a range of representative organizational settings. Lastly, and perhaps 
most obvious, this investigation was limited to only two of the three stakeholder groups 
identified. As explained, tlie training provider group did not have a population sufficient to 
survey and compare relative to the size of the other two groups. 

Even given these limitations, however, the results obtained might be best viewed as a 
useful beginning for continuing exploration beyond the scope of the present research. Also, in 
addition to these quantitative results, a good number of written comments were also supplied. 
These were useful to more thoroughly understand the quantitative results. They are discussed 
next. 



Written Comments 

In addition to responding to the quantitative survey items, respondents also provided a 
higher than anticipated volume of written comments. The written comments were especially 
useful in assisting in the overall interpretation and meaningfulness of the survey data. Using a 
similar coding analysis approach as described for the previous qualitative research, these 
comments were content analyzed^*. This section presents the results of these analyses. 



All qualitative analyses were performed with the aid of QSR NUD-IST (4.0) software. 





Training results 

Comments were extracted from the completed surveys and transcribed for coding and 
analysis. Because the comments were provided by respondents in the context of a particular 
survey section (and were therefore bounded by the questions being asked) two broad coding 
categories were defined to correspond respectively with the first two sections of the survey on 
training results and training program evaluation. Several start codes were defined to correspond 
with the research questions and conceptual categories initially established for the analysis of the 
quantitative results (see Tables 3 and 5 respectively). In addition, several codes were added 
during analysis to accommodate other comments not related to the start codes defined. Both the 
start and add-on codes are shown in Appendix D. While both managers and non-managers 
commented similarly in many areas, several differences also emerged from the analysis. 

Both managers and non-managers commented about training results in common terms of 
employee and customer satisfaction, product development, and business results. Examples of 
typical comments about training results made by managers included the following: 

• A happy, productive, effective employee producing a high-quality effective product. 
Poor, ineffective, training is more a dissatisfier than good training is a satisfier; 
continuous employee development; development of skills needed for future growth of 
[the company] 

• Happy and productive employees that are knowledgeable in both technical skills, 
process skills best practices, and people skills 

• Improved effectiveness; improved ESAT [employee satisfaction] and CSAT 
[customer satisfaction] 

Non-managers provided similar comments related to employee and customer satisfaction: 

• Increased awareness of products, processes; CSAT as well as ESAT 

• Improve employee satisfaction 

• A satisfied employee 

• Effectively trained and satisfied participants 

Both groups also recognized training’s role in keeping up with technological, and industry- 
specific change. Related to this managers commented that training should result in 

• increased productivity, quick familiarity with product and work environment; allow 
designers to stay current with leading-edge technology advances and practices 

• a program which can adopt quickly to changing needs 

Perhaps because they are most directly affected by job-specific knowledge requirements, non- 
managers cited training results mostly in relation to their own jobs in relation to product 
development. In their view effective training results in 

• employees knowledgeable in their field; continuing training is necessary to stay on the 
cutting edge [it] makes employees productive as fast as possible; improves employee- 
customer relations; leads to product innovations 

• [broadening] employee knowledge as well as to enhance their effectiveness 

• [a] continuously update [of] employees knowledge (since the technology is evolving so 
fast) 

• employees who are effective in their current positions and up-to-date with the changing 
world. [Also serves as] a method for employees to improve professionally and improve 
their knowledge 

• employees that can adapt to a rapidly changing business environment; employees that are 
willing/capable of anticipating future customer requirements 

• [developing employee awareness of] technological domain trends 

• [developing employee awareness of] market trends, new technology 
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Such comments about “keeping up” also extended into those specifically about helping newly- 
hired employees learn sufficiently fast to become effective on the job. The following manager 
comment is illustrative; 

[The training program should produce] new hires effectively trained and confident to act 

independently more often; reducing other staff s time in providing assistance; increasing 

quality of their output sooner; connecting them to their environment faster 
However, beyond these comments, non-managers tended to describe training results more 
specifically in terms of their own professional development. They also tended to better articulate 
training’s relationship to the roles and importance of non-management employees (such as 
designers and engineers) than did managers. Non-managers specifically connected improved 
professional knowledge and skills with the ultimate benefit of the company. In this regard they 
saw training results in terms of 

• More productive employees; better specific or general knowledge of the product or 
company; better well-being for the employee which pays off for the [company] and its 
shareholders 

• Improving the productivity of the employees; increase ESAT; open-up new job- 
related opportunities for employees; help retain talented employees; improve the 
overall competitiveness of [the company] 

• Quality and productivity; higher morale; employee effectiveness and better 
performance; better communication of complex ideas and designs; fewer conflicts due 
to ignorance or differences in background; a professional skilled workforce that is 
keeping up with changes in technology 

• A satisfied employee who can effectively perform assigned tasks; the outcome of 
assignments will be received by everyone involved management , employee, and 
customer increased satisfaction and sense of self which translates to bottom-line 
productivity 

While both groups also recognized strictly personal training results, non-managers emphasized 
such results as illustrated by the following comments; 

• Specific knowledge usable for employee for job at hand; general knowledge of the 
employee’s field; personal skills usable on the job; general skills and knowledge of 
personal value to the employee 

• a program which provides a balance between technical and non-technical courses as 
well as allows professional development in non-work related areas 

• being kept happy with up-to-date technical training in their area as well as personal- 
interest courses 

In contrast to such comments, managers made reference to organizational, business, and program 
objectives; 

• Achieving a specific objective, be it specific skills required for a job or softer skills 
which can be applied generally; in the case of [our corporate division] these objectives 
should be driven by both business needs as well as organizational [needs] 

• Does the program meet the objectives? Do the target audience accrue the 
value/knowledge desired? Evaluation involves collecting evaluation data, analyzing 
and presenting results to trainers and sponsors; the results should be used to determine 
if business requirements [are met] 

• Metrics against objectives (defects, productivity) 

Although non-managers also made reference to training’s relation to such objectives (as described 
above) this group emphasized training results most related to job, professional, and career 
development. Both the written comments from part 1 (training evaluation) and those from part 2 
(training evaluation) reflect the quantitative findings described. Comments from survey part 2 are 
discussed next. 
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Training Evaluation 



Written comments obtained from both groups regarding training program evaluation were 
more general. In contrast to part 1 of the survey on stakeholder views of training results, there 
was a tone in the written responses that “training evaluation is not my specialty so please don’t 
expect me to comment too knowledgeably or authoritatively in this area.” The following 
manager comments support this view: 

• I never heard of [training program evaluation] before today, therefore, I am not 
familiar with its objectives. 

• What is this “training evaluation” thing? Are you talking about a pre-existing 
process, some new process, or simply the act of asking people to assess the quality of 
courses they have taken? I have assumed it is the latter. 

The following comment provided by a non-manager also illustrates this view: 

• You guys are the professionals; I have no idea how training programs should be 
evaluated. 

Also, as pointed out in connection with the survey response demographics, such comments may 
help to explain the very slight decrease in the number of respondents for part 2 of the survey 
compared with part 1. One individual, a non-manager who completed the first but not the second 
part quantitative items of the survey, commented in the part 2 section: 

Was this survey [training program evaluation]? If so, I guess I should have answered the 
questions, if not, then I’m not familiar with [training program evaluation] and thus not in 
position to answer. 

Nevertheless, many respondents did provide comments which serve to shed light upon and further 
explain the response patterns observed. The next section describes stakeholder comments 
provided in connection with the purposes of training evaluation. 

Purposes 

Neither group favored exclusively formative or summative evaluation purposes based on 
their written comments. Rather, both indicated a balanced view between evaluation for training 
program improvement and Judgment. Representative manager comments supporting formative 
evaluation purposes included 

• [training program evaluation] results in long-term curriculum improvements. 

• Training evaluations should be used to continuously improve training. 

Similarly, non-managers commented that training program evaluation 

• [should] improve the quality of the existing programs; lead to the design of new 
programs to meet the new needs of the [lines of business] 

• [leads to] a continuous improvement in the material presented in the training sessions 
and the presenters of the material 

• result[s] in improved training programs 

• [leads to] improved training programs; facilitate[s] employee course decisions 

• result[s] in better training program 

• [should be used to] improve [the] training program 

Manager comments indicating a more summative posture included evaluation purposes to 

• result in an accurate audit of the value currently being delivered by training 

• ensure [the] program meets stated objectives 

• identity the value to the company 

Non-manager comments supporting this more summative perception of evaluation purposes 
included suggestions that training program evaluation 

• results in determining program effects and goals 

• [results in the] elimination or revision of ineffective programs 
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• [serves to] prune courses — discontinue some, add others 

• [determines] whether the program is effective 

• [serves to] evaluate the usefulness of the program 

• [serves to assess] the effectiveness of the training program 

• [allows evaluators] to see what has been gained from the program 

• [helps to] remove ineffective training 

• [serves to] to determine the degree of success 

• [helps] to determine if the program should continue 

• [helps] to decide which courses to keep 

In addition to comments regarding generally formative and summative purposes, 
miscellaneous other comments focused on assessing instructor competence, facilitating team 
building, and benchmarking the training program against those in other firms. Beyond these 
perceived evaluation purposes, one non-manager implied a certain professional faith in training 
professionals (and perhaps reinforced a general view that respondents do indeed recognize 
training and its evaluation as special areas of expertise outside their own) by indicating the 
purpose of training program evaluation to be simply “whatever uses the training team chooses.” 
Comments made in connection with training evaluation processes are presented and discussed 
next. 

Processes 

Written comments were also included pertaining to the training evaluation process. 
Corresponding to the survey, these comments were focused on the identification of (1) who 
should be involved in the process, (2) data required, and (3) the reporting of evaluation results. 
The following sections present written comments made by both managers and non-managers 
about these areas of the evaluation process. 

Involvement. 

Both groups made comments to indicate favoring a range of involvement in the training 
program evaluation process. In these comments many respondents specified a more precise 
hierarchical relationship for the involvement of immediate line management, that is, one level up 
from the training participant. Managers specifically mentioned the involvement of both the 
“immediate manager” of the training participant, as well as, “executive/senior management.” 
Non-managers commented on the involvement of the following: 

• senior business line managers 

• managers 

• trainees’ managers 

• immediate manager 

• next level of management 

• business line managers for future ventures 

• immediate manager 

The following comments on the involvement of training participants were also included: 

• involve trainees perception of usefulness of training as applicable to their 
responsibilities 

• to determine. . . if the program is meeting employee’s expectations and needs 

• feedback from trainees that are attempting to apply what they have learned 

Of all comments provided by both managers and non-managers, only a few comments (by non- 
managers) implicating the involvement of training specialists were included: 

• help course developers improve course content and presentation to meet the needs of 
the trainees and line management 
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• [training] specialists 

• individuals who deliver training 

In addition to suggesting the involvement of each of the three stakeholder groups several 
respondents identified several other internal and external groups. Managers suggested the 
involvement of “universities and other companies” as well as “external consultants who are 
experts in their area.” Non-managers mentioned the involvement of 

• individual contributors [ICs] 

• customers 

• auditors [ISO and quality] 

• observation by external expert [individuals and groups] 

• trainers (the ones training) 

• subject matter experts (SMEs) 



Data. 

Substantially fewer comments were made about the kinds of data required for the training 
program evaluation process. In addition to the data formats explicitly stated in the survey items 
included, both groups suggested several otlier forms of data related to the annual employee 
performance review process. Several non-managers further commented on the need to obtain 
feedback data from course instructors as well as training program participants as follow-up and 
performance data 1 month after the completion of a program course. An interesting comment 
was also made by a manager who simply stated that “some data are essential, but no individual 
data listed above are essential.” This individual did not elaborate on what these “essential” data 
might be. 

Reporting. 

Comments about training program evaluation reporting included suggestions to report 
results to both management and non-management groups in the line organization as well as 
instructors of program courses. While, relative to tlie otlier categories for which it was possible to 
supply written comments, very few comments were included about tlie report audience, both 
managers and non-managers emphasized that any such results should be made widely available to 
any interested individual or group in the organization. One manager simply commented that 
training program evaluation results should be made available to “all stakeholders.” Echoing this 
view non-managers suggested availability to 

• all interested parties 

• whomever applicable (based on need) 

• anyone in the organization 

• [the organizational] community 

As an external reporting audience, one non-manager also suggested that training program 
evaluation be made available to “customers.” 

Consequences 

A relatively few original comments were offered by either group pertaining to the 
consequences of training program evaluation. There are at least three possible reasons for this. 

The first of these is related simply to respondent fatigue as the comments fields for this section of 
the survey were all located on the third page (of the four-page suryey). A second possible 
explanation is the general lack of perceived respondent expertise in evaluation. As discussed 
earlier, several respondents indicated that they felt “unqualified” to add extensive comments 
based on their limited evaluation background. A third reason is that the eight part 2 survey items 
(2.18 through 2.25) effectively covered and exhausted the gamut of consequence possibilities. 
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Particularly in connection with the first reason (fatigue) by the time most respondents reached the 
additional comments section they may well have been eager to reach the last page to complete the 
survey. Nevertheless, some additional comments were provided here. These suggested largely 
instrumental evaluation consequences, for example, for course and program improvement as well 
as to assess whether training participant needs were met. 

Discussion and Limitations of Qualitative Comments 

Rather than promote selective interpretation of the quantitative results discussed, the 
written comments presented and discussed should be viewed to augment those results. Brief 
comments were liberally provided by both managers and non-managers. With a few exceptions, 
most of these can be placed in one or several of the following categories: (1) comments that 
restated or reiterated one or more of the survey items (e.g., training should contribute to customer 
or employee satisfaction), (2) comments expressing a lack of expertise perceived as a requirement 
to respond “correctly” to the items in the second part of the survey (e.g., I don’t know much about 
evaluation so don’t expect my responses to be very meaningful), (3) specific suggestions for 
course or program improvement (e.g., course notes should be made available to participants 
before they attend the course). Nevertheless, along with some of the more original comments 
which spoke more directly to the research questions, taken as a whole the comments should be 
viewed as a valuable addition to the quantitative results in providing additional infomiation about 
respondent perceptions of training results and evaluation. 

Overall Limitations 

Both general and specific limitations have been discussed regarding this investigation. 

As described above most of these concern the quantitative aspects of reliability, validity, internal 
consistency, and generalizability of the findings. Regarding the qualitative data collected through 
the written comments, some additional limitations should be mentioned. First, the written 
comment sections of the survey were necessarily restricted. While many respondents did choose 
to use the limited space provided to supply such comments. By design, however, these were 
relatively brief and focused on the topical areas specified (training results and evaluation). While 
one respondent actually included a separate sheet of typed comments, this was clearly the 
exception rather than the norm. Hence, because this investigation was concerned primarily with 
quantitative data collection, most of its limitations revolve around the quantitative limitations 
discussed above. 

A second overall limitation concerns the scope. As discussed, sampling for the 
investigation was done in a single division that is part of a much larger total corporate 
organization. While this division seems intuitively representative of both other divisions within 
the company (and perhaps the company itself), only repeated measures involving larger samples 
can substantiate further generalization of results. As mentioned, this is certainly the case for 
generalizations beyond the case organization itself. 

Summary and Conclusions 

Even with the limitations mentioned, the results obtained in this research have served to 
supplement and extend findings from the previous two research phases. In terms of the predicted 
relationships and research questions posed, perceptual variation regarding training results was 
found to be moderate. These findings generally correspond well with both the pattern matching 
relationships (Michalski & Cousins, in press) and the qualitative results (Michalski, 1998). While 
group views regarding evaluation were observed to depend moderately on training results in 
terms of employee satisfaction, each group emphasized different aspects of this construct. 

Both the quantitative variation indicated about training results (in terms of employee 
satisfaction) and training program evaluation (in terms of instrumental-conceptual use) were also 
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further explained by the written comments obtained. Managers perceived the importance of 
employee satisfaction more in terms of productivity and project requirements, whereas non- 
managers clearly expressed employee satisfaction more specifically in terms of their own 
professional and career development. This outcome can be further used to explain variation 
among the two groups based on each group’s tendency to use evaluation both instrumentally and 
conceptually in ways that are congruous with their respective group’s perceptions of training 
results. This conclusion is supported by both the univariate and the multivariate differences 
described. 

Beyond this the results have also enabled further generalization to the case organization. 
As anticipated, based on both the original predicted relationships and the results of the previous 
two research phases, the limited variation found can be understood in terms of both groups co- 
location within the core function of the organization. Organizationally, and from a business 
perspective, both groups are much more closely aligned and united in their focus on meeting 
external customer and market requirements. The training provider group (not included in the 
survey because of their relatively small numbers) on the other hand generally focuses on 
satisfying both of these core stakeholder groups as internal organizational clients. 
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Appendix A. Survey Cover Letter and Instrument 
Survey Cover Letter 



To: SONET/INM Employees May 6, 1998 

Subject: Training Program Evaluation Survey 

During the past week you should have received a postcard informing you of the Training 
Program Evaluation Survey. This survey is enclosed and your cooperation to complete it 
is appreciated. 

We are aware that you are routinely asked to complete various satisfaction surveys, however, this 
one is substantially different. In addition to gauging your satisfaction with training in 
SONET/INM, the survey data will also be used to advance knowledge about training evaluation 
as part of a doctoral (Ph.D.) research project through the University of Ottawa. Greg Michalski, a 
regular full-time employee with the Advanced Technology Training group, is the doctoral 
candidate performing this research. Data for this comprehensive case study have been collected 
for about the last year. To date many of your peers and co-workers have contributed valuable 
ideas to the project. The survey enclosed represents the culmination of these efforts. 

As one of over 400 randomly selected regular full-time employees being asked to 
participate your response to this survey is crucial because your views will serve to 
represent those of all division 1 employees.. 

Pilot tests were used both to improve the quality and to minimize the time required to 
complete the form. Most employees who pilot tested the survey completed it in about 20 
minutes. All responses are completely anonymous. Pooled survey results will be made 
available directly to you through the Transmission Training web site (web URL address) 
after all surveys are returned. 

If you have questions about the survey or research project, please contact Greg at 
(internal extension). This research project has been approved by the Ethics Committee at 
the Faculty of Education, University of Ottawa, Chair Professor A. Giroux (562-5800 
x4066). 

Please complete the enclosed survey form and return it using interoffice mail (Training 
Evaluation Survey, internal mail address) in the envelope provided within one week of receipt. 

Your participation is greatly appreciated. 

Many thanks, 

(signed) 

Vice President Principle Researcher 



Enc. Training Program Evaluation Survey 
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Appendix A (continued) 



Survey Instrument 

7RAININGPRCX3RAMEVALUAT10NSURVEY(OpticalNetworks) 

This questionnaire has four parts. Part I seeks your views on the results of all training (as an 
int^rated program) available to you as an employee. Part II asks about the evaluation of thi 
training. Part III asks you to rate your satisfaction with the Transmission Training program. 
Part IV asks a few (important) employee background questions about you. 

Please con^leteand return thequestionnairein theenvelopeprovided(WDLN-1 Mailroom, Attn: Training 
Evaluation Survey) within ONE week of receipt If you have questions, please call ESN 393-3736. 



PARTI: Training Results 



ForeachofthefoIIowingstatementsindicateyourviewbyblackeningONEoption: ^ dark pen / f 

® @ . 

Strongly Disagree Neither Agree Strongly ' 

Disagree Agreenor Ag.^ Notllkethis: 

Disagree 



I . I The main purpose of training should be to improve employee productivity. 

1 .2 Effective training ultimately improves business profitability. 

1 .3 Product quality can be improved substantially through employee training. 

1 .4 The availability of high-quality training leads to increased employee satisfaction. 

1. 5 Only job-specific training should be provided to employees. 

1 .6 The main purpose of training should be to improve customer satisfaction. 

1 .7 Training program objectives should be derived directly from business objectives. 

1 .8 The main purpose of training should be to develop employees professionally. 

1 .9 Effective training should help designers to anticipate customer requirements. 

I . lO Product time-to-market can be reduced substantially through employee training. 

1 . 1 1 All training provided to employees should have a positive "return on investment". 

1.12 Effective training should improve employee understanding of product development 

1.13 Employee promotion and employee training are directly related. 

1 . 1 4 Effective training should directly support our customer business objectives. 

1 . 1 5 Effective training ultimately improves employee-customer relations. 

1 . 1 6 Training is the best way to make new employees productive as quickly as possible. 

1 . 1 7 Employee pay and employee training are directly related. 

1. 1 8 Effective training ultimately leads to product innovation. 

1. 1 9 Effective training should result in the immediate use of knowledge on the job. 

1. 20 An effective training program attracts or retains the most talented employees. 
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Appendix A (continued) 

Part I WRITTEN COWIVENTS: Print belowany comments you have about training resu 
Hint: Thisfield is machine-readable. NeatlyprintedCAPITALlettasaremosteasilyrecognized. 
TRAINING RESULTS: An effective training program should result in... 




PARTII: Training Program Evaluation (TPE) 



Part II seeks your views on the evaluation of all training available to employees as an integrated program 
(consisting of all courses, self-paced learning rrxxlules, computer-based training). 

For each ofthefollowingstatementsindicateyourviewbyblackeningONEoption: 

@ © 0 0 (g) ..... * 

Strongly Disagree Neither Agree Strongly circl^ like this. 

Disagree Agree nor Agree A/Of like this: gf 

Disagree 



2. 1 The main purpose of training program evaluation (TPE) is to improve the program. 

2.2 The main purpose of TPE is to judge whether the program is meeting its objectives. 

2.3 The main purpose of training program evaluation is OTHER (PRINT NEATLY below): 



2.4 Training specialists should be actively involved in performing the TPE. 

2.5 Trainees (training recipients) should be actively involved in performing the TPE. 

2.6 Training sponsors (line management) should be actively involved in performing the TPE. 

2.7 External evaluation experts should be actively involved in performing the TPE. 
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.2.9 Questionnaire data are essential for TPE. 

2. 1 0 Interview data are essential for TPE. 

2. 1 1 Focus group data are essential in TPE. 

2. 12 Knowledge test data from training participants are essential in TPE. 
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Appendix A (continued) 



PARTH (cont.): Training Program Evaluation (TPE) 



2. 1 3 OTHER data are essential in TPE (PRINT NEATLY below): 



2. 14 TPE results should be reported directly to training specialists. 

2.15 TPE results should be reported directly to trainees (training recipients). 

2.16 TPE results should be reported directly to training sponsors (line management). 

2. 1 7 TPE results should be reported directly to OTHER (PRINT NEATLY below): 



2. 1 8 TPE results should be used mainly as a basis for future decisions about the program. 

2.19 TPE results should be used mainly to develop new knowledge about program effects. 

2.20 TPE results should be used mainly to comply with quality standards such as ISO audits. 

2.2 1 TPE results should be used mainly to routinely monitor program activities. 

2.22 TPE results should be used mainly to determine trainee satisfaction with the program. 

2.23 TPE results should be used mainly to determine manager satisfaction with the program. 

2.24 TPE results should be used mainly to determine whether program goals are met. 

2.25 TPE results should be used mainly to assist trainees in selecting courses. 

2.26 OTHER TPE results (PRINT NEATLY below): 
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Part 11 WRITTEN COMMENTS: Print below any comments you have about training program evaluation. 

Hint: Thisfieldisrmchne-reackible. NecUlyprintedCAPFrALlettersarernosteasilyrecognized 

TRAININGEYALUnON: Training program evaluation should (accomplish, involve, result in).. 
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Appendix A (continued) 



Partin asksyoutorateyoursatisfaction with the Transmission (Optical Networks) Trainin 
Program. The following statements refer specifically to training offered in the Transmissio 
Tra/h/hgProgramdescribedat(http;//47.97.96.115/Transmission/). 



PARTIII:Transmi ssionTra iningProgramSatisfac1 



Foreach of the followingstatementsindicateyourviewby blackening ONEoption: 





@ ® ® ® ® Rll circles like this: 


• 












Strongly Disagree Neither Agree Strongly Wof like this: 

Disagree Agreenor Agree 














Disagree 
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3.2 


The program offers me adequate training to develop in my career. 
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The program offers me adequate personal development training. 
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I consider the training program to be an asset to me as an employee. 
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The time I spend taking training offered in the program is worthwhile to me. 
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The program meets my needs as an employee in SONET/INM. 
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3.7 


Training offered in the program helps me to understand customer requirements. 
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Training offered in the program helps me to understand product development process d 
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The training program contributes to product design quality in SONET/INM. 
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I would like the program to offer me formal competency certification options. 
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Part IV asks you to provide important demographic infonnation about yourself. All responses are strictly 
anonymous and confidential. This infonnation is needed to analyze the overall results of the survey. 
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PARTIV: Backgroundinformation 



4.1 Please indicate your job role (blacken the appropriate circle). 

O non-manager q manager 
^ 2 How long have you been working with [the company]? 

O less than 1 year Ql to 5 years Q 5 to 10 years Q 10 to 20 years Q 20 years or more 

4.3 Please select the job category that best describes the work you do. 

Please PRINT NEATLY in area below: 

O software o hardware o verification o captive office o other c> ’ 

4.4 Please select the product you are most closely associated with. 

Please PRINT NEATLY below: 

OOC-3 qOC-12 oOC- 48 oOti;-192 qINM O other 



Thank you very much for your participation. Please mail your form back in the envelope provided 

[Return Address] 
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Appendix B. Survey Response Demographics. 



Response by Responsibility 

(Manager/Non-Manager) 




Response by Experience 

(time in company) 



20 years or more 
15 (5%) 



less than 1 year 
38 (14%) 



5 to 1 0 years 
40 (14%) 




1 0 to 20 years 
56 (20%) 



1 to 5 years 
131 (47%) 
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Appendix B (continued) 



Response by Product Group 



other 
30 (11%) 



OC-3 

37(13%) 



OC-12 
39 (14%) 




OC-48 
51 (18%) 



OC-192 
123 (44%) 



Note: Product development groups are organized by transmission data capacity. Optical carrier (OC) refers 
to the fiber optic data transmission. OC-1 is the basic SONET (synchronous optical network) transmission 
rate of 51.840 Mbits per second. OC-3, 12 ,..., 192 are multiples of this. OC-192, also referred to as “high- 
capacity” transport, is capable of transmission rates of lOGbits per second. 



Response by Job Category 



verification 



captive office 



other_ 
23 (8%) 



hardware 
82 (29%) 




software 
169 (60%) 
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Appendix C. Factor Analysis Results 

Factor Analysis of Part I Items on Training Program Results 



Rotated Component Matrix? 


Component 




1 


2 3 


4 


5 


pay 


.788 








promote 


.758 








custreqs 


.538 








innovat 


.490 








emp_cust 


.460 


.450 






roi 




.749 






know_use 




.685 






custobj 


.403 


.514 






devjiroc 




.457 






new_emp 










qual 




.772 






ttm 




.640 






empjirod 






.660 




job_spec 






.588 




busjirof 




.423 


.579 




csat 






.527 




lr_objs 




.423 


.502 




esat 








.667 


prof_dev 








.640 


attr_emp 








.637 



Extraction Method: Principal Component Analysis. 
Rotation Method: Varimax with Kaiser Normalization. 



Rotation converged in 22 iterations. 



Total Variance Explained 

Rotation Sums of Squared Loadings 

% of Cumulative 



Component 


Total 


Variance 


% 


1 


2.462 


12.311 


12.311 


2 


2.092 


10.460 


22.770 


3 


2.057 


10.284 


33.054 


4 


1.969 


9.846 


42.900 


5 


1.849 


9.247 


52.147 



Extraction Method: Principal Component Analysis. 
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Appendix C (continued) 



Rotated Component Matrij^ 



Component 




1 


2 


3 


4 


5 


6 


7 


aud_use 


.756 














mon_use 


.681 














mgrsatus 


.623 














trsatuse 
















rep_spon 




.745 












goal_use 




.587 












questnne 






.659 










dec_use 






.559 










trainee 






.533 










rep_trsp 






.513 










effctuse 
















intervw 








.811 








focusgrp 








.760 








knowtest 








.658 








tr_spec 










.745 






ext_invt 










.632 






sponsor 




.537 






.584 






formtv 












.834 




summtv 












.760 




rep_tran 














.826 


cseselus 














.595 



Extraction Method: Principal Component Analysis. 
Rotation Method: Varimax with Kaiser Normalization. 



Rotation converged in 9 iterations. 

Factor Analysis of Part II Items on Training Program Evaluation 



Total Variance Explained 



Rotation Sums of Squared Loadings 


Component 


Total 


%of 

Variance 


Cumulative 

% 


1 


5.1^6 


ro.286 




2 


2.028 


9.657 


19.943 


3 


2.026 


9.647 


29.590 


4 


1.961 


9.339 


38.929 


5 


1.823 


8.683 


47.612 


6 


1.683 


8.015 


55.627 


7 


1.463 


6.967 


62.594 



Extraction Method: Principal Component Analysis. 
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Appendix D. Code and Index Structure for Survey Written Comments 

(1) /tr-results: training program results 

(1 1) /tr-results/bus&mkt: business and market 

(1 2) /tr-results/CS AT: customer satisfaction 

(1 3) /tr-results/prod-dev: product development 

(1 4) /tr-results/emp-prod: employee productivity 

(1 5) /tr-results/tr-efficiency: training efficiency 

(1 6) /tr-results/ESAT: employee satisfaction 

(16 1) /tr-results/ESAT/general: general employee satisfaction 

(16 2) /tr-results/ESAT/prof-devlpt: employee professional/career development 

(16 3) /tr-results/ESAT/personal: employee personal development 

(17) /tr-results/other; other training results* 

(17 1) /tr-results/other/strat-knowledge: strategic knowledge* 

(17 2) /tr-results/other/org-goal&eff: organizational/company objectives/efficiency* 

(1 7 3) /tr-results/other/new hires: newly hired employees* 

(17 4) /tr-results/other/keeping up: keeping up with technology* 



( 2 ) 

(2 1 ) 

(2 1 1 ) 
(2 12 ) 
(2 13) 
(2 13 1) 
(2 13 2) 
(2 13 3) 
(2 2 ) 

(2 2 1 ) 

(2 2 11) 
(2 2 12 ) 
(2 2 13) 
(2 2 14) 
(2 2 15) 
(2 2 2 ) 

(2 2 2 1 ) 
(2 2 2 2 ) 
(2 2 2 3) 
(2 2 2 4) 
(2 2 2 5) 
(2 2 3) 

(2 2 3 1) 
(2 2 3 2) 
(2 2 3 3) 
(2 2 3 4) 
(2 3) 

(2 3 1) 

(2 3 2) 

(2 4) 

(2 5) 



/tr-eval: training program evaluation 
/tr-eval/purpose: training program evaluation purpose 
/tr-eval/purpose/formative: formative purposes 
/tr-eval/purpose/summative: summative purpose 
/tr-eval/purpose/other: other purposes of TPE 
/tr-eval/purpose/other/instructor: evaluate the instructor 
/tr-eval/purpose/other/team: team involvement 

/tr-eval/purpose/other/benchmark; compare with other companies and organizations 
/tr-eval/process: TPE processes 
/tr-eval/process/involvmt; involvement 
/tr-eval/process/involvmt/mgt: management 
/tr-eval/process/involvmt/other: other individuals or organizations 
/tr-eval/process/involvmt/trainees: trainee involvement 
/tr-eval/process/involvmt/tr-devlpers: training developers 
/tr-eval/process/involvmt/SME's: subject matter experts 
/tr-eval/process/data: data 

/tr-eval/process/data/MFA: annual employee performance appraisal record* 
/tr-eval/process/data/other: other data 
/tr-eval/process/data/instructor: instructor/sme* 

/tr-eval/process/data/timing: timing of data collection 
/tr-eval/process/data/emp-perf; employee performance* 

/tr-eval/process/reporting: evaluation reporting 
/tr-eval/process/reporting/mgt: management 
/tr-eval/process/reporting/other: others 
/tr-eval/process/reporting/instructors: instructors/sme's 
/tr-eval/process/reporting/trainees: trainees 
/tr-eval/consequence: TPE consequences 

/tr-eval/consequence/knowledge: increased or improved knowledge* 
/tr-eval/consequence/emp-needs: employee needs* 

/tr-eval/current probs: problems with current system* 

/tr-eval/courses: course- focused* 



*denotes add-on code developed after initial start codes were defined 



Appendix D (continued) 

Index tree for training results written comments 




(1) tr-results 





-> (1 6) ESAT; 



(1 5) tr-efficiency 

(1 4) emp-prod 
(1 3) prod-dev 
(1 2) CSAT 
(1 1) bus&mkt 



(1 6 3) personal 
-► (1 6 2) prof-devipt 
(16 1) general 
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Appendix D (continued) 

Coding index tree for evaluation written comments. 



(2 5) courses 
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